Undertaking rapid evaluations during the COVID-19 pandemic: Lessons from evaluating COVID-19 remote home monitoring services in England

Introduction Rapid evaluations can offer evidence on innovations in health and social care that can be used to inform fast-moving policy and practise, and support their scale-up according to previous research. However, there are few comprehensive accounts of how to plan and conduct large-scale rapid evaluations, ensure scientific rigour, and achieve stakeholder engagement within compressed timeframes. Methods Using a case study of a national mixed-methods rapid evaluation of COVID-19 remote home monitoring services in England, conducted during the COVID-19 pandemic, this manuscript examines the process of conducting a large-scale rapid evaluation from design to dissemination and impact, and reflects on the key lessons for conducting future large-scale rapid evaluations. In this manuscript, we describe each stage of the rapid evaluation: convening the team (study team and external collaborators), design and planning (scoping, designing protocols, study set up), data collection and analysis, and dissemination. Results We reflect on why certain decisions were made and highlight facilitators and challenges. The manuscript concludes with 12 key lessons for conducting large-scale mixed-methods rapid evaluations of healthcare services. We propose that rapid study teams need to: (1) find ways of quickly building trust with external stakeholders, including evidence-users;(2) consider the needs of the rapid evaluation and resources needed;(3) use scoping to ensure the study is highly focused;(4) carefully consider what cannot be completed within a designated timeframe;(5) use structured processes to ensure consistency and rigour;(6) be flexible and responsive to changing needs and circumstances;(7) consider the risks associated with new data collection approaches of quantitative data (and their usability);(8) consider whether it is possible to use aggregated quantitative data, and what that would mean when presenting results, (9) consider using structured processes & layered analysis approaches to rapidly synthesise qualitative findings, (10) consider the balance between speed and the size and skills of the team, (11) ensure all team members know roles and responsibilities and can communicate quickly and clearly;and (12) consider how best to share findings, in discussion with evidence-users, for rapid understanding and use. Conclusion These 12 lessons can be used to inform the development and conduct of future rapid evaluations in a range of contexts and settings.

Undertaking rapid evaluations during the COVID-19 pandemic: Lessons from evaluating COVID-19 remote home monitoring services in England.

Introduction: Rapid evaluations can offer evidence on innovations in health and social care that can be used to inform fast-moving policy and practise, and support their scale-up according to previous research. However, there are few comprehensive accounts of how to plan and conduct large-scale rapid evaluations, ensure scientific rigour, and achieve stakeholder engagement within compressed timeframes. Methods: Using a case study of a national mixed-methods rapid evaluation of COVID-19 remote home monitoring services in England, conducted during the COVID-19 pandemic, this manuscript examines the process of conducting a large-scale rapid evaluation from design to dissemination and impact, and reflects on the key lessons for conducting future large-scale rapid evaluations. In this manuscript, we describe each stage of the rapid evaluation: convening the team (study team and external collaborators), design and planning (scoping, designing protocols, study set up), data collection and analysis, and dissemination. Results: We reflect on why certain decisions were made and highlight facilitators and challenges. The manuscript concludes with 12 key lessons for conducting large-scale mixed-methods rapid evaluations of healthcare services. We propose that rapid study teams need to: (1) find ways of quickly building trust with external stakeholders, including evidence-users; (2) consider the needs of the rapid evaluation and resources needed; (3) use scoping to ensure the study is highly focused; (4) carefully consider what cannot be completed within a designated timeframe; (5) use structured processes to ensure consistency and rigour; (6) be flexible and responsive to changing needs and circumstances; (7) consider the risks associated with new data collection approaches of quantitative data (and their usability); (8) consider whether it is possible to use aggregated quantitative data, and what that would mean when presenting results, (9) consider using structured processes & layered analysis approaches to rapidly synthesise qualitative findings, (10) consider the balance between speed and the size and skills of the team, (11) ensure all team members know roles and responsibilities and can communicate quickly and clearly; and (12) consider how best to share findings, in discussion with evidence-users, for rapid understanding and use. Conclusion: These 12 lessons can be used to inform the development and conduct of future rapid evaluations in a range of contexts and settings.

Prehospital video triage of potential stroke patients in North Central London and East Kent: rapid mixed-methods service evaluation

BackgroundIn response to COVID-19, alongside other service changes, North Central London and East Kent implemented prehospital video triage: this involved stroke and ambulance clinicians communicating over FaceTime (Apple Inc., Cupertino, CA, USA) to assess suspected stroke patients while still on scene.ObjectiveTo evaluate the implementation, experience and impact of prehospital video triage in North Central London and East Kent.DesignA rapid mixed-methods service evaluation (July 2020 to September 2021) using the following methods. (1) Evidence reviews: scoping review (15 reviews included) and rapid systematic review (47 papers included) on prehospital video triage for stroke, covering usability (audio-visual and signal quality);acceptability (whether or not clinicians want to use it);impact (on outcomes, safety, experience and cost-effectiveness);and factors influencing implementation. (2) Clinician views of prehospital video triage in North Central London and East Kent, covering usability, acceptability, patient safety and implementation: qualitative analysis of interviews with ambulance and stroke clinicians (n = 27), observations (n = 12) and documents (n = 23);a survey of ambulance clinicians (n = 233). (3) Impact on safety and quality: analysis of local ambulance conveyance times (n = 1400;April to September 2020). Analysis of national stroke audit data on ambulance conveyance and stroke unit delivery of clinical interventions in North Central London, East Kent and the rest of England (n = 137,650;July 2018 to December 2020).Results(1) Evidence: limited but growing, and sparse in UK settings. Prehospital video triage can be usable and acceptable, requiring clear network connection and audio-visual signal, clinician training and communication. Key knowledge gaps included impact on patient conveyance, patient outcomes and cost-effectiveness. (2) Clinician views. Usability – relied on stable Wi-Fi and audio-visual signals, and back-up processes for when signals failed. Clinicians described training as important for confidence in using prehospital video triage services, noting potential for ‘refresher' courses and joint training events. Ambulance clinicians preferred more active training, as used in North Central London. Acceptability – most clinicians felt that prehospital video triage improved on previous processes and wanted it to continue or expand. Ambulance clinicians reported increased confidence in decisions. Stroke clinicians found doing assessments alongside their standard duties a source of pressure. Safety – clinical leaders monitored and managed potential patient safety issues;clinicians felt strongly that services were safe. Implementation – several factors enabled prehospital video triage at a system level (e.g. COVID-19) and more locally (e.g. facilitative governance, receptive clinicians). Clinical leaders reached across and beyond their organisations to engage clinicians, senior managers and the wider system. (3) Impact on safety and quality: we found no evidence of increased times from symptom onset to arrival at services or of stroke clinical interventions reducing in studied areas. We found several significant improvements relative to the rest of England (possibly resulting from other service changes).LimitationsWe could not interview patients and carers. Ambulance data had no historic or regional comparators. Stroke audit data were not at patient level. Several safety issues were not collected routinely. Our survey used a convenience sample.ConclusionsPrehospital video triage was perceived as usable, acceptable and safe in both areas.Future researchQualitative research with patients, carers and other stakeholders and quantitative analysis of patient-level data on care delivery, outcomes and cost-effectiveness, using national controls. Focus on sustainability and roll-out of services.Study registrationThis study is registered as PROSPERO CRD42021254209.FundingThis proj c was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research;Vol. 10, No. 26. See the NIHR Journals Library website for further project information.

Healthcare Leadership with Political Astuteness and its role in the implementation of major system change: the HeLPA qualitative study

BackgroundThe implementation of change in health and care services is often complicated by the ‘micropolitics’ of the care system. There is growing recognition that health and care leaders need to develop and use types of ‘political skill’ or ‘political astuteness’ to understand and manage the micropolitics of change.AimThe aim of this study was to produce a new empirical and theoretical understanding of the acquisition, use and contribution of leadership with ‘political astuteness’, especially in the implementation of major system change, from which to inform the co-design of training, development and recruitment resources.MethodsThe qualitative study comprised four work packages. Work package 1 involved two systematic literature reviews: one ‘review of reviews’ on the concept of political astuteness and another applying the learning from this to the health services research literature. Work package 2 involved biographical narrative interviews with 66 health and care leaders to investigate their experiences of acquiring and using political skills in the implementation of change. Work package 3 involved in-depth qualitative case study research with nine project teams drawn from three regional Sustainability and Transformation Partnerships operating in different English regions. Work package 4 involved a series of co-design workshops to develop learning materials and resources to support service leaders’ acquisition and use of political skills and astuteness.ResultsThe concepts of political skills and astuteness have had growing influence on health services research, yet these have tended to emphasise a relatively individualised and behavioural view of change leadership. The interview study suggests that, although leaders certainly use individual skills and behaviours when implementing change, change processes are contingent on local contextual factors and the patterns of collective action in the forms of interlocking constellations of political interactions. The in-depth case study research further shows these interactive, contingent and collective processes in the implementation of major system change. The study finds that major system change occurs over several linked stages, each involving particular controversies for which skills, strategies and actions are needed. Informed by these findings, and through a series of co-design workshops, the study has produced a set of resources and materials and a workbook to support individuals and project teams to acquire and develop political skill.LimitationsThe study was complicated by the COVID-19 pandemic and there were difficulties in recruiting in-depth cases for observational research, and also recruiting patient and community groups.ConclusionsHealth and care leaders can develop and use a range of skills, strategies and actions to understand and navigate the diverse interests that complicate change. Building on the literature, the study presents a novel empirical framework of these skills, strategies and behaviours, and shows how they are used in the implementation of major system change. This study concludes with a set of co-designed learning resources and materials to support future leaders to develop similar skills and strategies. Further evidence is needed on the contribution of the learning resources on leadership activities and to understand the contribution of political skills to other areas of service governance.Study registrationThis study is registered as researchregistery4020.FundingThis project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research;Vol. 10, No. 11. See the NIHR Journals Library website for further project information.

Healthcare Professional and Patient Perceptions of Changes in Colorectal Cancer Care Delivery During the COVID-19 Pandemic and Impact on Health Inequalities.

BACKGROUND: The COVID-19 pandemic changed the way in which people were diagnosed and treated for cancer. We explored healthcare professional and patient perceptions of the main changes to colorectal cancer delivery during the COVID-19 pandemic and how they impacted on socioeconomic inequalities in care. METHODS: In 2020, using a qualitative approach, we interviewed patients (n = 15) who accessed primary care with colorectal cancer symptoms and were referred for further investigations. In 2021, we interviewed a wide range of healthcare professionals (n = 30) across the cancer care pathway and gathered national and local documents/guidelines regarding changes in colorectal cancer care. RESULTS: Changes with the potential to exacerbate inequalities in care, included: the move to remote consultations; changes in symptomatic triage, new COVID testing procedures/ways to access healthcare, changes in visitor policies and treatment (e.g., shorter course radiotherapy). Changes that improved patient access/convenience or the diagnostic process have the potential to reduce inequalities in care. DISCUSSION: Changes in healthcare delivery during the COVID-19 pandemic have the ongoing potential to exacerbate existing health inequalities due to changes in how patients are triaged, changes to diagnostic and disease management processes, reduced social support available to patients and potential over-reliance on digital first approaches. We provide several recommendations to help mitigate these harms, whilst harnessing the gains.

The impact of post-hospital remote monitoring of COVID-19 patients using pulse oximetry: A national observational study using hospital activity data.

Background: There was a national roll out of 'COVID Virtual Wards' (CVW) during England's second COVID-19 wave (Autumn 2020 - Spring 2021). These services used remote pulse oximetry monitoring for COVID-19 patients following discharge from hospital. A key aim was to enable rapid detection of patient deterioration. It was anticipated that the services would support early discharge, reducing pressure on beds. This study is an evaluation of the impact of the CVW services on hospital activity. Methods: Using retrospective patient-level hospital admissions data, we built multivariate models to analyze the relationship between the implementation of CVW services and hospital activity outcomes: length of COVID-19 related stays and subsequent COVID-19 readmissions within 28 days. We used data from more than 98% of recorded COVID-19 hospital stays in England, where the patient was discharged alive between mid-August 2020 and late February 2021. Findings: We found a longer length of stay for COVID-19 patients discharged from hospitals where a CVW was available, when compared to patients discharged from hospitals where there was no CVW (adjusted IRR 1·05, 95% CI 1·01 to 1·09). We found no evidence of a relationship between the availability of CVW and subsequent rates of readmission for COVID-19 (adjusted OR 0.97, 95% CI 0.91 to 1·03). Interpretation: We found no evidence of early discharges or changes in readmissions associated with the roll out of COVID Virtual Wards across England. Our analysis made pragmatic use of national-scale hospital data, but it is possible that a lack of specific data (for example, on which patients were enrolled and on potentially important confounders) may have meant that true impacts, especially at a local level, were not ultimately discernible. It is important that future research is able to make use of better quality - preferably linked - data, from multiple sites. Funding: This is independent research funded by the National Institute for Health Research, Health Services & Delivery Research program (RSET Project no. 16/138/17; BRACE Project no. 16/138/31) and NHSE&I. NJF is an NIHR Senior Investigator.

Socioeconomic differences in help seeking for colorectal cancer symptoms during COVID-19: a UK-wide qualitative interview study.

BACKGROUND: COVID-19 has led to rapid changes in healthcare delivery, raising concern that these changes may exacerbate existing inequalities in patient outcomes. AIM: To understand how patients' help-seeking experiences in primary care for colorectal cancer symptoms during COVID-19 were affected by their socioeconomic status (SES). DESIGN AND SETTING: Qualitative semi-structured interviews with males and females across the UK, recruited using purposive sampling by SES. METHOD: Interviews were carried out with 39 participants (20 higher SES; 19 lower SES) who contacted primary care about possible symptoms of colorectal cancer during COVID-19. Data were analysed using framework analysis followed by comparative thematic analysis to explore differences between groups. RESULTS: Three themes were identified with differences between SES groups: 1) how people decided to seek medical help through appraisal of symptoms; 2) how people navigated services; and 3) impact of COVID-19 on how patients interacted with healthcare professionals. The lower SES group expressed uncertainty appraising symptoms and navigating services (in terms of new processes resulting from COVID-19 and worries about infection). There was also potential for increased disparity in diagnosis and management, with other methods of getting in touch (for example, email or 111) taken up more readily by higher SES patients. CONCLUSION: The findings suggest that COVID-19 exacerbated disparities between higher and lower SES participants. This study raises awareness around challenges in help seeking in the context of the pandemic, which are likely to persist (post-COVID-19) as healthcare systems settle on new models of care (for example, digital). Recommendations are provided to reduce inequalities of care.

Patients' experiences of, and engagement with, remote home monitoring services for COVID-19 patients: A rapid mixed-methods study.

INTRODUCTION: Remote home monitoring models were implemented during the COVID-19 pandemic to shorten hospital length of stay, reduce unnecessary hospital admission, readmission and infection and appropriately escalate care. Within these models, patients are asked to take and record readings and escalate care if advised. There is limited evidence on how patients and carers experience these services. This study aimed to evaluate patient experiences of, and engagement with, remote home monitoring models for COVID-19. METHODS: A rapid mixed-methods study was carried out in England (conducted from March to June 2021). We remotely conducted a cross-sectional survey and semi-structured interviews with patients and carers. Interview findings were summarized using rapid assessment procedures sheets and data were grouped into themes (using thematic analysis). Survey data were analysed using descriptive statistics. RESULTS: We received 1069 surveys (18% response rate) and conducted interviews with patients (n = 59) or their carers (n = 3). 'Care' relied on support from staff members and family/friends. Patients and carers reported positive experiences and felt that the service and human contact reassured them and was easy to engage with. Yet, some patients and carers identified problems with engagement (e.g., hesitancy to self-escalate care). Engagement was influenced by patient factors such as health and knowledge, support from family/friends and staff, availability and ease of use of informational and material resources (e.g., equipment) and service factors. CONCLUSION: Remote home monitoring models place responsibility on patients to self-manage symptoms in partnership with staff; yet, many patients required support and preferred human contact (especially for identifying problems). Caring burden and experiences of those living alone and barriers to engagement should be considered when designing and implementing remote home monitoring services. PATIENT OR PUBLIC CONTRIBUTION: The study team met with service users and public members of the evaluation teams throughout the project in a series of workshops. Workshops informed study design, data collection tools and data interpretation and were conducted to also discuss study dissemination. Public patient involvement (PPI) members helped to pilot patient surveys and interview guides with the research team. Some members of the public also piloted the patient survey. Members of the PPI group were given the opportunity to comment on the manuscript, and the manuscript was amended accordingly.

The impact of remote home monitoring of people with COVID-19 using pulse oximetry: A national population and observational study.

BACKGROUND: Remote home monitoring of people testing positive for COVID-19 using pulse oximetry was implemented across England during the Winter of 2020/21 to identify falling blood oxygen saturation levels at an early stage. This was hypothesised to enable earlier hospital admission, reduce the need for intensive care and improve survival. This study is an evaluation of the clinical effectiveness of the pre-hospital monitoring programme, COVID oximetry @home (CO@h). METHODS: The setting was all Clinical Commissioning Group (CCG) areas in England where there were complete data on the number of people enrolled onto the programme between 2nd November 2020 and 21st February 2021. We analysed relationships at a geographical area level between the extent to which people aged 65 or over were enrolled onto the programme and outcomes over the period between November 2020 to February 2021. FINDINGS: For every 10% increase in coverage of the programme, mortality was reduced by 2% (95% confidence interval:4% reduction to 1% increase), admissions increased by 3% (-1% to 7%), in-hospital mortality fell by 3% (-8% to 3%) and lengths of stay increased by 1·8% (-1·2% to 4·9%). None of these results are statistically significant, although the confidence interval indicates that any adverse effect on mortality would be small, but a mortality reduction of up to 4% may have resulted from the programme. INTERPRETATION: There are several possible explanations for our findings. One is that CO@h did not have the hypothesised impact. Another is that the low rates of enrolment and incomplete data in many areas reduced the chances of detecting any impact that may have existed. Also, CO@h has been implemented in many different ways across the country and these may have had varying levels of effect. FUNDING: This is independent research funded by the National Institute for Health Research, Health Services & Delivery Research programme (RSET Project no. 16/138/17; BRACE Project no. 16/138/31) and NHSEI. NJF is an NIHR Senior Investigator.

Remote home monitoring (virtual wards) for confirmed or suspected COVID-19 patients: a rapid systematic review.

BACKGROUND: the aim of this review was to analyze the implementation and impact of remote home monitoring models (virtual wards) for confirmed or suspected COVID-19 patients, identifying their main components, processes of implementation, target patient populations, impact on outcomes, costs and lessons learnt. METHODS: we carried out a rapid systematic review on models led by primary and secondary care across seven countries (US, Australia, Canada, The Netherlands, Ireland, China, UK). The main outcomes included in the review were: impact of remote home monitoring on virtual length of stay, escalation, emergency department attendance/reattendance, admission/readmission and mortality. The search was updated on February 2021. We used the PRISMA statement and the review was registered on PROSPERO (CRD: 42020202888). FINDINGS: the review included 27 articles. The aim of the models was to maintain patients safe in the appropriate setting. Most models were led by secondary care and confirmation of COVID-19 was not required (in most cases). Monitoring was carried via online platforms, paper-based systems with telephone calls or (less frequently) through wearable sensors. Models based on phone calls were considered more inclusive. Patient/career training was identified as a determining factor of success. We could not reach substantive conclusions regarding patient safety and the identification of early deterioration due to lack of standardized reporting and missing data. Economic analysis was not reported for most of the models and did not go beyond reporting resources used and the amount spent per patient monitored. INTERPRETATION: future research should focus on staff and patient experiences of care and inequalities in patients' access to care. Attention needs to be paid to the cost-effectiveness of the models and their sustainability, evaluation of their impact on patient outcomes by using comparators, and the use of risk-stratification tools.

The implementation of remote home monitoring models during the COVID-19 pandemic in England.

BACKGROUND: There is a paucity of evidence for the implementation of remote home monitoring for COVID-19 infection. The aims of this study were to identify the key characteristics of remote home monitoring models for COVID-19 infection, explore the experiences of staff implementing these models, understand the use of data for monitoring progress against outcomes, and document variability in staffing and resource allocation. METHODS: This was a multi-site mixed methods study conducted between July and August 2020 that combined qualitative and quantitative approaches to analyse the implementation and impact of remote home monitoring models developed during the first wave of the COVID-19 pandemic in England. The study combined interviews (n = 22) with staff delivering these models across eight sites in England with the collection and analysis of data on staffing models and resource allocation. FINDINGS: The models varied in relation to the healthcare settings and mechanisms used for patient triage, monitoring and escalation. Implementation was embedded in existing staff workloads and budgets. Good communication within clinical teams, culturally-appropriate information for patients/carers and the combination of multiple approaches for patient monitoring (app and paper-based) were considered facilitators in implementation. The mean cost per monitored patient varied from £400 to £553, depending on the model. INTERPRETATION: It is necessary to provide the means for evaluating the effectiveness of these models, for example, by establishing comparator data. Future research should also focus on the sustainability of the models and patient experience (considering the extent to which some of the models exacerbate existing inequalities in access to care).